What is self-hosted scraping with managed proxy rotation?

Short answer: Self-hosted scraping with managed proxy rotation is the hybrid model where you run the scraper binary on your own server but route its outbound fetches through a third-party residential or datacenter proxy pool (Bright Data, Oxylabs, IPRoyal, Smartproxy). You keep ownership of the scraping logic and data; you outsource only the IP rotation problem.

Why split the stack this way

Running your own residential proxy network is uneconomic for most teams — the legal and supply-chain overhead of acquiring clean residential IPs is a full business by itself. Conversely, handing the entire scrape to a SaaS vendor means giving up control of the response body, the extraction prompt, and the audit trail. The hybrid model keeps the expensive-to-build proxy pool with a specialist while keeping the easy-to-build (and sensitive) scraping logic in-house.

How the configuration looks

A managed proxy is just an HTTP proxy URL with credentials: http://user-session-abc:pass@brd.superproxy.io:22225. You pass that URL to the scraper per request, and the scraper forwards every outbound fetch through it. Sticky sessions, geo-targeting, and country-level rotation are controlled by encoding session tokens into the username — every major provider supports this pattern. Rotation happens entirely inside the proxy network; the scraper doesn't have to know.

How fastCRW does it

fastCRW accepts a per-request proxy parameter on /v1/scrape and /v1/crawl. Pass the proxy URL — including credentials — and the Rust client uses it for the outbound fetch; sticky-session usernames work transparently. The same self-hosted binary can be pointed at a different proxy per request, so you can route by target hostname, by tenant, or by geo. See scraping and configuration for the full request shape.

Frequently asked

Which managed proxy providers work with self-hosted scrapers?

Any provider that exposes a standard HTTP or SOCKS5 proxy endpoint works — Bright Data, Oxylabs, IPRoyal, Smartproxy, NetNut, and most others. The scraper does not need a provider-specific SDK; the proxy URL with credentials is enough.

Does the proxy provider see the scraped page content?

The proxy provider sees the destination URL and the encrypted TLS traffic, just like any HTTP forward proxy. They can see request volume and target hostnames; they cannot read response bodies inside HTTPS without active TLS interception, which reputable providers do not do.

Can I rotate proxies per request without restarting the scraper?

Yes. A self-hosted scraper that accepts a per-request proxy parameter — fastCRW does — can route every call through a different upstream proxy URL without any process restart, which is essential for high-volume rotating residential pools.

← CRW docs home